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This invention relates to a system and 
method for reducing blocking artifacts intro- 
duced by current compression algorithms that 
compress images as independent blocks of pix- 
els. Preferably, the methods of the inven- 
tion include determining block-to-block dif- 
ferences in edge pixels or in overall intensities 
between adjacent pixel blocks, selecting pixel 
blocks for post-processing that appear to be in 
relatively featureless regions of the image, in- 
terpolating the block-to-block edge differences 
into an error correction matrix, and then sub- 
tracting the error correction matrix from the 
original pixel block. These methods are prefer- 
ably implemented in special software routines 
that execute on micro-processor based systems 
or on digital signal processor based systems op- 
timized for image decoding. 
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BACKGROUND OF THE INVENTION 
. 1. Field of the Invention 

The present invention relates to systems and methods for post-processing 
decompressed images in order to minimize perceptual artifacts due to prior image 
5 compression, and in particular to minimize blocking artifacts. 

2: Description of the Related Art 

Many important image compression methods process images as independent 
blocks of pixels. For example, such families of compression standards as JPEG, MPEG, 

10 H.320, and so forth, specify a step involving discrete cosine transformation ("DCT") of 
independent, non-overlapping 8X8 blocks of pixels in the source image followed by 
quantization of the resulting transform coefficients. See, e.g., Jack, 1996, Video Demystified, 
HighText Interactive Inc., San Diego, CA. The quantized transform coefficients are 
transmitted from a transmitter-encoder to a receiver-decoder. Such transformation and 

15 quantization together achieve compression by exploiting the significant correlations that 
typically occur between the values of pixels in 8X8 blocks, but result in loss of image 
information ("lossy" compression), the coarser the quantization the greater the loss. 

Decompressing images so compressed, which necessarily involves steps of 
dequantization and inverse DCT of the received quantized coefficients in order to derive a 

20 received decompressed image, can lead to what are called herein "blocking artifacts" in the 
following manner. In certain areas of a received image, the quantization errors introduced can 
become especially apparent and even objectionable. Especially, in regions where the image is 
fairly smooth, with little high spatial frequencies components', errors in the low spatial 
frequency components can make the individual, independent 8X8 blocks perceptually 

25 apparent. This is especially so if low frequency components, which smoothed the block-to- 
block boundaries in the source image, are set to zero. 

Several methods of reducing such blocking artifacts are available in the current 
state of the art. Simple lowpass filtering applied to the decompressed image can blur blocking 
artifacts and reduce their prominence to some extent, but it necessarily leads also to an overall 
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degraded sharpness in the image. Blocks, can be overlapped in the source image in order to 
redundantly encode block-to-block boundaries, but at the cost of decreased compression and 
increased required communication bandwidth. 

Further, Pennebaker et al„ 1993, JPEG Still Image Compression, Van Nostrand 
5 Reinhard, chap. 16, discloses JPEG block smoothing by fitting quadratic surfaces to the 
average values of pixels (equivalent to the "DC", or lowest order, transform coefficient) in 
adjacent blocks, a computationally complex process. Lakhani, 1996, "Improved Image 
Reproduction from DC Components", Opt. Eng. 35:3449-2452, discloses equations for 
predicting low frequency transform coefficients from DC coefficients that are improved from 

10 those in the JPEG standard. Finally, Jeon et aL, 1995, Blocking Artifacts Reduction in Image 
Coding Based on Minimum Block Boundary Discontinuity, Proc, SPIE 2501 : 1 89-209, 
discloses a complex and computationally expensive iterative method for zeroing block 
boundary discontinuities. 

Importantly, all current art methods appear to achieve blocking artifact 

15 reduction by in one fashion or another performing versions of spatial low-pass filtering. These 
current art methods also all suffer from one or more additional problems, such as producing 
overall image degradation, limiting image compression, failing to explicitly address the 
perceptual aspects of blocking artifacts, requiring excessive computational resources, and so 
forth. 

20 What is needed, therefore, is a method and system for postprocessing 

decompressed images which is computationally efficient, avoids spatial low-pass filtering, 
does not produce image degradation, has no effect on compression, and, most importantly, 
minimizes the perceptual aspects of blocking artifacts. 

25 SUMMARY OF THE INVENTION 

The objects of the present invention are to provide systems and methods for 
post-processing decompressed images in order to minimize blocking artifacts and which 
overcome the above identified problems in the current art. To this end, the invention provides 
a method and a system as defined in the independent Claims. The dependent Claims define 

30 advantageous embodiments. 

Fundamentally, these objects are achieved by methods which achieve blocking 
artifact reduction by correcting the surface defined by the pixel values in a block of pixels with 
"bending", "lilting", or "twisting" deformations in order to more closely match pixel-value 
surfaces of adjoining pixel blocks. Since low-pass filtering is avoided by such surface 
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deformations, the corrections added to the pixel values by the methods of this invention more 
closely match the actual errors and artifacts introduced by the image blocking process. 

, In detail, these objects are achieved by determining an 8X8 matrix of correction 
values for each processed 8X8 pixel block in an image. The correction matrices are then added 
5 to the pixel blocks in order to derive post-processed pixel blocks with minimized blocking 
artifacts. The resulting pixel values of the corrected pixel blocks blend with pixel values of 
adjacent blocks, also typically corrected, in a perceptually smooth manner with minimum 
block-to-block artifacts. The .correction matrices are derived from differences between values 
of pixels along the edges of a block to be post-processed and pixels along the edges of the four 

10 orthogonally adjacent pixel blocks. Alternatively, the correction matrix is derived from zero- 
frequency ("DC M ) transform coefficients of a pixel block to be post-processed and the four 
adjacent pixel blocks. 

The 8X8 matrix of correction values is either derived directly according to a 
preferred entirely spatial-domain interpolation, or derived indirectly by an alternative 

15 computation from a smaller 4X4 spatial-domain intermediary error correction matrix. Direct 
and inverse transforms of the intermediary correction matrix to and back from a frequency 
domain accomplish smooth interpolation of the smaller intermediary matrix to an 8X8 matrix 
of correction values. Preferably, decoded blocks are selected for post-processing principally in 
relatively flat or featureless image regions. Such image regions are most likely to have 

20 perceptually apparent blocking artifacts. 

BRIEF DESCRIPTION OF THE DRAWING 

Other objects, features and advantages of the present invention will become 
apparent upon perusal of the following detailed description when taken in conjunction with the . 
25 appended drawing, wherein: 

Figs. 1A-B illustrate exemplary embodiments of systems of the present 

invention; 

Figs. 2A-B illustrate exemplary pixel blocks; 

Fig. 3 illustrates exemplary embodiments of methods of the present invention; 
30 . Figs. 4A-C illustrate exemplary spatial weighting configurations; 

. Fig. 5 illustrates a test image; 

Fig. 6 illustrates the test image of Fig. 5 with a maximum of blocking artifacts; 

and * • ■„ . . 
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Fig. 7 illustrates the result of post-processing the image of Fig. 6 according to 
an embodiment of the present invention. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
5 In the following, exemplary embodiments of systems of the present invention 

are first described followed by detailed descriptions of embodiments of the methods of the 
present invention. Finally, an example of applying an embodiment of the present invention to 
a test image with a maximum of blocking artifacts is presented. 

10 Preferred Systems Embodiments 

Fig. 1 A generally illustrates an embodiment of a system according to the 
present invention, represented within box 4, in conjunction with a conventional image decoder 
system, including input 1, decoder 2, and image buffer 3. 

The present invention is adaptable to decoded images that were encoded by any 
15 process that divides an original image into non-overlapping, rectangular blocks of pixels, and 
independently encodes each such rectangular block. In the following, the invention is 
described with respect to such conventional families of compression standards as JPEG, 
MPEG, and H.320, according to which an original image is divided into non-overlapping and 
independent 8X8 blocks of pixels, which are encoded as quantized DCT coefficients. 
20 The invention is most effective when the images are highly compressed, and, 

accordingly, when quantization of the DCT coefficients is aggressive or coarse. Exemplary of 
such compression levels is compression to 0.25 or less bits of luminance information per pixel. 

Encoded images are input at 1 to the conventional decoder system illustrated. 
Decoder 2 then performs conventional decoding of the input images according to the encoding 
25 method, and stores decoded images temporarily in image buffer 3. In the absence of the 

present invention, the images are directly output from the image buffer. The present invention 
adds the processing elements and steps represented within box 4 to such a conventional 
decoder. 

The general processing operations performed by the present invention are next 
30 generally described with details following subsequently. In first processing operation 5, 
(determine block differences) quantities representative of blocking artifacts in a decoded or 
decompressed image, are determined for all blocks from various aspects block-to-block pixel 
value differences. Preferably, four such representative quantities are determined for each pixel 
block. In second processing operation (select blocks to process) 6, the individual blocks to be 
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post-processed by the present invention are selected heuristically in view of the determined 
block-to-block differences. In one embodiment, if any of these differences exceed a threshold 
value, the block is not processed; alternatively, if the differences exceed the threshold, their 
values are set down to that threshold. This step reflects the discovery of the inventor that large 
5 block-to-block differences are likely caused by structure actually present in the original image, 
while small block-to-block differences typically reflect blocking artifacts present in relatively 
featureless regions on the original image. These should be corrected. Finally, in third 
processing operation (determined 8x8 correction matrix) 7, for each post-processed 8X8 block, 
an 8X8 matrix of correction values is determined according to the alternative embodiments to 

10 be described subsequently. The correction values are determined to smoothly link the post- 
processed block to its orthogonally-adjacent blocks. Processing of the blocks in the image 
buffer can be sequential or in parallel according to the available hardware. 

Finally, the correction matrix and the original decoded block are added by adder 
8 and stored back in image buffer 3. When all blocks in the image buffer have been processed, 

15 a corrected image with minimal blocking artifacts is output at 10. 

Optionally, the DC coefficients (i.e., zero frequency coefficients) transmitted in 
the encoded images can be used in processing step 5 to determine the block-to-block 
differences. In this case, it is advantageous for the decoder 2 to make these coefficients 
available to determine block differences unit 5 over connection 9, which can include a 

20 coefficient buffer as needed. 

This system can be implemented in various hardware configurations that will be 
apparent to one of average skill. For example, the individual image processing operations 
represented in Fig. 1 A can be implemented with individually dedicated hardware components. 
However, it is currently preferable that the processing operations of the system be 

25 implemented by one or more special software routines running on general purpose hardware, 
perhaps optimized for image decoding, such as that illustrated in Fig. IB. Fig. IB illustrates 
one or more processors 1 1 for decoding images and performing the operations of the present . 
invention, one of more RAM modules 12 for storing image data and/or program instructions, 
optionally one or more ROM modules 13 for storing program instructions, one or more I/O 

30 interface devices 14 for communicating with other systems, and one or more busses 15 for 
connecting these individual components. Advantageously, the processors include one or more 
digital signal processors ("DSP M ), such as the TM-1000 type DSP (Philips Electronics North 
America Corp.) or the TMS-3000 type DSP (Texas Instruments, Inc.). 



WO 00/33254 5 PCT/EP99/08924 _ 

"... 6 

In the preferred embodiment where the system processing operations are 
implemented in software, the present invention further comprises computer readable media on 
which are recorded or encoded program instructions for causing processors to perform the 
processing operation of the system. Such media can include magnetic media, such as floppy 
5 discs, hard discs, tapes, and so forth, optical media, such as CD-ROMS, and other media 
technologies usable in the art; 

Preferred Method Embodiments 

Preferred and alternative embodiments of processing operations 5*7 (Fig. 1A) 

10 . are next described in detail. First, determination of block-to-block pixel differences, 
processing operation 5 (Fig. 3), which according to the present invention are taken to be 
representative of blocking artifacts, is described with reference to the candidate pixel blocks 
illustrated in Figs. 2A-B. Although these figures and the subsequent discussion treats the 
common case of 8X8 pixel blocks, one of average skill will immediately appreciate how to 

15 modify the methods to be described in case rectangular pixel blocks of sizes other than 8X8 
are used in a particular compression method. 

For candidate, central pixel block C in Fig. 2A, block-to-block pixel differences 
are determined with respect to the four, orthogonally-adjacent pixel blocks sharing edges with 
candidate, central block C. These blocks are labeled W (to the west of C), N (to the north of 

20 C), E (to the east of C), and S (to the south of C) in Fig. 2A. Diagonally adjacent pixel blocks 
are not directly considered. Fig; 2B illustrates pixel blocks C, N, E, W, and S in more detail in 
the case where the relevant image compression methods use 8X8 pixel blocks. Edge pixels for 
all these blocks are indicated in standard matrix notation. Also indicated are the DC 
coefficients (zero frequency coefficients) for these blocks (Cdc. Ndc* Edo Wdc. Sqc). It is 

25 well known that the DC coefficients are simply averages of the values of all the pixels in a 
block, and may be obtained directly from decoder 2 or may be computed as part of step 5. 

With reference to Fig. 2B, a first method for determining block-to-block pixel 
differences computes the average differences between pixels along each edge of block C and 
pixels along adjacent edges of blocks N, E, W, and S. Four quantities are returned, each 

30 reflecting the block-to-block differences along an edge of central block C. This preferred 

method, directly and with minimum computation, returns values representative of pixel-value 
discontinuities at the boundaries of a pixel block. The following equations are representative 
of this method, where the four quantities N, E. W, and S are average edge-adjacent pixel 
differences along the north, east, west, and south edges of central block C, respectively. 
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N= 0J25*FA*{(N7O + N7 I +N72 + N 7 3+.N74 + N75 + N76 + N7^ 

+ C03 + C(>4 + C05 + C()6 + C07)} 

5 E =* 0.125*FA*{(Eoo + E l0 + E20 + E 30 + E40 + E 50 + Eoo + E 70 ) - (C 07 + Cn + C27 + 

C37 + C47 + C57 + C67 + C77)} 

W= 0J25^A*((Wot + W, 7 ^ 

C20 + C30 + C40 + C50 + C60 + C70)} 

10 

S= 0.125*FA*{(Soo + Soi+So2 + So3 + S W + So5 + So6 + So7)r(C 7 0 + C71 +C72 + 

C73 + C74 + C75 + C76 + C77)} 

Factor FA is a multiplicative weighting factor for scaling these edge differences. It is 
15 described subsequently in more detail in conjunction with second weighting factor EB. 

A second alternative method for determining block-to block pixel differences 
employs DC coefficients, where available. This method returns four quantities representative 
of the differences in average intensity between central block C and its four orthogonally- 
adjacent surrounding blocks, N, E, W, andS. The following equations are representative of 
20 this method. 

N = FA*{N DC -C DC }; E = FA*{E DC -C DC } 

. WsFA*{Wk-Cdc); S=FA*{Sdc-Cdc} 

25 

Finally, a third alternative method captures increased detail of block-to-block 
pixel differences along block edges by computing sub-edge differences^ The methods above, 
which determine only four quantities, one for each edge of a central block, necessarily 
represent, therefore, only average differences along an entire edge. However, perceptual 
30 blocking artifacts can also reflect variations in block-to-block pixel differences along each 
edge, as well as simply block-to-block average differences. The third alternative method 
captures such variations in intensity by computing more than one quantity for each edge, each 
quantity representing the differences in one or more pairs of edge-adjacent pixels of central 
block C, instead of merely the average of all pixels along an edge. In a case where averages of 
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the differences of two adjacent pairs of pixels are computed, the following equations are 
representative of the third alternative. 

A = 0.250*FA'*{(W,7 - Co) + (W07 - Coo) + (N70 - Coo) + (N 7 , - C 0I )} 

5 . • 

B= 0.500*FA*{(N 72 -C 02 ) + (N 73 -Co 3 )} 

C = 0.500*FA*{(N 74 - C04) + (N75 - Cos)} 

10 D = 0.250*FA*{(N 76 - Q»j + (tin -On) + (Eoo - C 07 ) + (E01 - C 17 )} 

E= 0.500*FA* { (E 20 - Cr,) + (E30 - C 37 )} 

F= 0.500*FA*{(E4o-C4 7 ) + (E5o-C 57 )} 

15 . 

G= 0.250*FA*{(E 6 o-C 67 ) + (E 7 o-C 77 ) + (So 7 -C 77 ) + (So6-C 76 )} 

H= 0.500»FA*{(So5-C 75 ) + (So4-C 7 4)} 

20 1= 0.500*FA*{(So3-C 73 ) + (So2-C 72 )} 

J= 0.250*FA*{(Soi-C 7I ) + (Soo-C 7 o) + (W 77 -C 7 o) + {W 67 -C 6 o)) 

K= 0.500*FA*{(W5 7 -C5o) + (W4 7 -C4o)} 

25 

L= 0.500*FA*{(W 37 -C3o) + (W 27 -C 20 )} 

For, illustrative purposes, quantities J, I, H, and G represent differences between pair of pixels 
in the outlined sub-blocks 16, 17, 18, and 19, respectively. Of course, a finer representation 
30 could be used, for example one in which a quantity is computed from the difference of each 
pair of edge-adjacent pixels. Alternately, a coarser.representation could be used by combining 
certain of the above quantities. For example, the following combined and coarser quantities 
can be used. 
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B' = C' = 0.500*(B + C); E' = F = 0.500*(E + F) 
r = H' = 0,500*(I + H); L' = K' = 0.500*(L + K) 

Other similar alternatives of increased or decreased coarseness will be apparent to one of skill 
in the art. 

Turning to the next processing operation, operation 6 of Fig. 3, blocks are 
selected for post-processing in view of heuristic block selection criteria that depend on the 
block-to-block pixel differences determined above. The overall goal and purpose of the system 
and method of the present invention is to achieve perceptually improved images, and not to 
post-process images merely to achieve certain mathematical effects or consistency without 
regard to perceptual changes. Accordingly, the methods of the present invention include 
heuristic block selection criteria, adjustable weighting factors for block-to-block pixel 
differences, and computational alternatives that can be adjusted or selected in particular cases 
to obtain maximum perceptual improvements, in general or with respect to image classes or 
interest, within the processing resources available. 

In detail, block selection criteria are derived from the discovery of the inventor 
that in many cases block-tp-block pixel differences represent image structures in a source 
image whose correction can result in perceptually objectionable image alterations. Such 
differences are likely to represent real image structure where the differences along one or more 
edges are particularly "large". In contrast, where block-to-block differences are "small", the 
actual image is likely to be relatively flat or featureless, i.e., devoid of particularly apparent 
structure. In such regions small blocking artifacts are most perceptually apparent, and, 
therefore, should be corrected according to the present invention. Additionally, "small" block- 
to-block pixel difference are also a likely indicator of differences due only to blocking 
artifacts. 

; "Small" is advantageously measured in view of a particular image compression 
algorithm as that degree of block-to-block pixel differences which are of a magnitude likely to 
be due to quantization errors in the low frequency transform coefficients, in particular as that 
degree of blpck-to-block pixel differences resulting from zeroing of low frequency transform 
coefficients as a result of coarse quantization and subsequent dequantization. A "small" 
difference is taken to be of the magnitude of this quantization error, or in other words 
preferably approximately 2-4 times such quantization error. A "large" block-to-block pixel 
difference, on the other hand, is preferably 5 or more times such quantization error. Therefore, 
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block with "small" block-to-block pixel differences are post-processed, while block with large 
differences are not. 

In particular, for JPEG or MPEG, which use DCT transformation of 8X8 
blocks, applied to pixels represented by 8 bits (values between 0 and 255), "small" is 
5 preferably a block-to-block difference of less than 5, and more preferably less than 2. A "large 
difference is preferably 6, or more preferably 8, or more. Exact values are advantageously 
selected to achieve the maximum perceptual improvement. 

In a preferred block selection criterion, any pixel block having any block-to- 
block difference with an adjacent block which is greater than the sielected threshold is not 

10 processed Alternatively, all blocks can be processed but any block-io-block difference 

exceeding the selected threshold value is set back to and limited by the threshold value. This 
latter alternative is advantageous in order to correct blocks with one side bordering actual 
image structure and another side bordering flat or featureless image regions. 

In a further alternative, where all transform coefficients are available from a 

15 decoder, blocks with little AC (non-zero frequency) energy (determined, for example, as the 
sum of the squares of the transform coefficients) are chosen for processing. For example, 
blocks with no more than 5%, or more preferably no more than 3%, of their energy in the AC 
terms are selected for processing. 

For the blocks selected in processing operation 6 and using the block-to-block 

20 pixel differences determined in processing operation 5, processing operation 7 (Fig. 1 A) next 
determines matrices of error correction values which are added to the processed blocks by 
adder 8 to finally arrive at corrected blocks. The error correction matrices are the same size as 
the pixel blocks. Corrected images result from processing all pixel blocks in the images. Fig. 3 
illustrates processing operation 7 in more detail as having two principal alternatives, preferred 

25 operation T and alternative operation 7". Generally, according to preferred operation 7*, the 
block-to-block pixel differences are directly converted into an 8X8 (or other size) error 
correction matrix. According to alternative operation 7", the differences are first converted 
(20) into a 4X4 intermediate error matrix, which is then smoothly expanded into a final 8X8 
error correction matrix by a transform (21) to a 4x4 frequency domain matrix, an embedding 

30 (22) in an 8x8 matrix, and an inverse transform (23) to an 8x8 spatial matrix. 

In more detail, preferred operation T interpolates the error correction matrix 
elements from the block-to-block pixel differences determined along each edge. This 
interpolation is done in a dimension independent manner, in that each matrix element has 
independent, weighted contributions from each determined block-to-block pixel difference, 
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and each determined block-to-block pixel difference contributes independently only to matrix 
elements in that row or column of that particular pixel difference. For example, in case of a 
value representing the average difference of edge-adjacent pixels along an entire edge, the 
value is interpolated along all rows or columns perpendicular to that edge. 
5 The weights of the pixel differences have chosen a spatial configuration and a 

selected overall multiplicative weight, FA*FB. The spatial configuration is chosen, limited by 
constraints, in order to result in a maximum perceptual improvement. One constraint is that the 
maximum spatial weight occur adjacent to the block edge of that block-tq-block difference to 
be interpolated. Another constraint is that the sum of the weights be zero in order that the 

10 average brightness of the pixel block does not change. A final heuristic constraint is that a 

difference at one edge of a pixel block coupled with another difference of the same size but ," 
different sign at the opposite edge (along a row or column) should interpolate to a uniform 
gradient between the two edges. 

Fig. 4C illustrates the spatial configuration weights for creating the 8X8 error 

15 correction matrix of the preferred alternative. The maximum weight occurs next to the 

interpolated block-to-block pixel difference. The sum of the weights is clearly zero. Finally, a 
unit positive pixel difference at a left edge together with a unit negative pixel difference at a 
right edge is interpolated by the spatial configuration of Fig. 4C into the preferred linear 
gradient of Fig. 4D. One of average skill will understand how to expand or compress this 

20 preferred spatial configuration for other pixel array sizes. 

The overall multiplicative factor, FA*FB, is also selected, limited by 
constraints, in order to result in a maximum perceptual improvement. The overall factor is 
divided between a weight factor for the block-to-block pixel difference, FA, and a weight 
factor for the error correction matrix, FB. According to one constraint, the overall factor, 

25 FA*FB, is less then or equal 0.500 so that edges between adjacent corrected pixel blocks are 
not over corrected, that is so that corrected pixel blocks have their block-to-block edge 
differences reduced without changing the direction of that edge difference. According to 
another overlapping constraint, the overall factor is less than 0.500 because, in the presence of 
pixel-value gradients, a certain block-to-block edge difference can be an actual image feature 

30 and should not be eliminated. There is also a weighting factor present for normalizing the 
maximum of the spatial configuration weights to 1.00. 

In preferred embodiments, FA is taken as 1.0, so that the block-to-block pixel 
differences represent unsealed pixel values. Second, FB is advantageously taken as 0.375 as 
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this factor gives good perceptual results and is rapid to compute by shifts and additions 
without multiplications. 

Accordingly, in a preferred embodiment of processing operation 7', the error 
correction matrix is interpolated from four block-to-block pixel differences, one for each edge 
5 of the pixel block to be post-processed, according to the following equations, where FA = 1.0 
in the determination of the N, E, W, and S differences and FB = 0.375*(1 .0/4.0). (1/0/4.0 is the 
spatial configuration weight normalization factor.) The N. E, W. and S input difference values 
are preferably determined as the average differences of edge-adjacent pixels described above, 
or alternative as the difference in the DC coefficients, also described above. The following 
10 equations defining this matrix are exemplary.. 

matrix[0][0] = FB*{ 4*N +4*W } 

matrix[0][l] = FB*{ 4*N-E +2*W } - 

matrix[0][2] = FB*{ 4*N-2*E } 
15 matrix[0][3]=FB*{ 4*N-2*E-W } 

matrix[0][4]=FB*{ 4*N -E -2*W } 

matrix[0]t5]=FB*{ 4*N -2*W } 

matrix[0][6]=FB*{ 4*N + 2*E-W } 
. matrix[0][7]=FB*{ 4*N + 4*E ) 
20 

matrix{l][0] = FB*{ 2*N +4*W-S} 
matrix[l][l]=FB*{ 2*N-E +2*W-S } 
matrix! 1] [2] =FB*{ 2*N-2*E -S) 
matrix[l][3]=FB*{ 2*N-2*E-W -S } 
25 matrix[l][4]=FB*{ 2*N-E -2*W-S } 
matrix[l][5] = FB*{ 2*N -2*W-S } 
matrix[l][6] = FB*{ 2*N + 2*E-W -S } 
matrix[l][7) = FB*{ 2*N + 4*E -S } 

30 matrix[2][0) = FB ,,, { +4*W-2*S) 

matrix[2][l] = FB*{ -E +2*W-2*S} 

matrixt2][2] = FB*{ - 2*E -2*S} 

matrix[2j[3) = FB*{ -2*E-W -2*S} 

matrix[2][4] = FB*{ -E -2*W-2*S) 
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The output is the 8X8 error correction matrix. 

It will be apparent how to alter these equations where any pixel block edge is 
associated with more that a single block-to-block pixel difference parameter. 

Turning to the alternative embodiment, generally this embodiment determines 

20 an error correction matrix which has a size that is smaller than the size of a pixel block and 
then expands this smaller error correction matrix to the size of the pixel block. The smaller 
error correction matrix preferably has a size that is a rational fraction of the size of the pixel 
block, e.g., a 4X4 size when the pixel block size is 8X8. The smaller error correction matrix is 
preferably expanded in a smooth fashion. For example, this expansion can be done by 

25 transforming to a transform domain followed by inverse transforming, with higher frequency 
coefficients set to zero, from that transform domain to a matrix the size of a pixel block. 

In more detail, this alternative embodiment, processing operation 7" of Fig 3, is 
illustrated as commencing with operation 20, determination of a 4X4 error matrix. In a 
preferred embodiment of this alternative method, this 4X4 matrix is determined similarly to 

30 the 8X8 matrix discussed above, that is matrix elements are interpolated from the block-to- 
block pixel differences determined along each edge in a dimension independent manner, such 
that each matrix element has independent, weighted contributions from each perpendicularly- 
related block-to-block pixel difference. These pixel differences weights also have a chosen 
spatial configuration and a selected overall multiplicative weight, FA*FB. The spatial 
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configuration is preferably chosen according to the principles discussed above. Fig. 4A 
illustrates a preferred spatial weight configuration having elements that sum to zero and which 
results in a linear gradient between equal but opposite differences at each edge of the 4X4 
matrix, as illustrated in Fig. 4R. The multiplicative weights are preferably selected as 
5 discussed above. Therefore, it is again preferred that FA = 1 .0 and FB = 0.375. Again, the N, 
E, W, and S input block-to-block pixel differences are preferably determined as the average 
differences of edge-adjacent pixels described above, or alternative as the differences in the DC 
coefficients, also described above. 

The following equations are exemplary of this preferred embodiment for 
10 defining the 4X4 matrix. FB equals 0.375 multiplied by 1.0/3.0, the spatial configuration 
normalization factor. 

matrix[0][0] = FB*{ 3*N + 3*W } 
matrix[0][l] = FB*{ 3*N -W-2*E} 
15 matrix[0][2]=FB*{ 3*N - 2*W-E } 
matrix[0][3]=FB*{ 3*N . + 3*E) 

matrix[l][0] = FB*{-N -2*S + 3*W. } 

matrix[I][l] = FB*{-N -2*S-W -2*E} 

20 matrix[l][2]=FB*{-N -2*S-2*W-E } 

matrix[l][3]=FB*{-N -2*S + 3*E} 

matrix[2][0] = FB*{-2*N-S +3*W ) 

matrix[2][l]=FB*{-2*N-S - W -2*E} 

25 matrix[2][2]=FB*{-2*N-S -2*W-E } 

matrixf2][3] = FB*{-2*N-S +3*E) 

matrix[3][0J = FB*{ +3*S + 3*W } 

matrixt3][l] = FB*( +3*S-W -2*E} 

30 matrix[3][2]=FB*{ +3"'S-2*W-E } 

matrix[3][3]=FB*{ +3*S +3*E) 



In another embodiment, this 4X4 matrix can be determined from pixel sub-edge 
differences, which reflect finer structure of the block-to-block pixel differences along an edge. 
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Exemplary of such sub-edge differences are the quantities A-L discussed above. In one 
alternative, these sub-edge difference can be placed around the edges of the 4X4 matrix, one 
sub-edge difference being placed in the 4X4 matrix at a position corresponding to the position 
of original pixels in the original 8X8 matrix. The central four matrix elements can be set to 
5 zero. The following equations are representative of this alternative, with FB being preferably 
0375. 

matrix[O][0]=FB*A; matrix [0][1]=FB*B; matrix [0][2]=FB*C 
matrix[0][3]=FB*D; matrix [1)[3]=FB*E; matrix [2][3]=FB*F 
10 matrix[3][3]=FB*G; matrix [3][2]=FB*H; matrix [3][1]=FB*I 
matrix[3][0]=FB*J; matrix [2][0]=FB*K; matrix (1][0]=FB*L 
matrix[l][l]=matrix[l][2]=matrixt2][l]=matrix[2][2] = 0.0 

Alternatively, the inner four matrix elements can be interpolated from the edge elements in a 
15 dimension independent manner using the spatial configuration weights of Fig. 4A. 

The next processing operations of alternative T expand the 4X4 matrix into an 
8X8 matrix in a smooth manner, that is introducing only a minimum of higher spatial 
frequency components into the final 8X8 matrix. In step 21, the 4X4 matrix is transformed 
into a 4X4 matrix in a suitable frequency domain. A DCT transform is preferred, but this 
20 invention is adaptable to other transforms, such as the computationally inexpensive Hadamard 
transform. In step 22, the 4X4 frequency domain matrix is embedded in an 8X8 frequency 
domain matrix with all the remaining elements set to 0. Alternatively, certain elements in the 
4X4 matrix, such as the bottom-most row and right-most column, may also be set to zero. 
Thereby the higher frequency components represented by these matrix elements are zero. 
25 Finally, in step 23, the 8X8 frequency domain matrix is inverse transformed into an 8X8 

spatial domain error correction matrix, which is used in the subsequent steps of this invention 
just as the 8X8 matrix of the preferred alternative is used. Preferably, the inverse transform is 
. the same as the forward transform, that is an inverse DCT is used in step 23 if a DCT is used 
instep 21. 

30 The 8X8 error correction matrix resulting from either the preferred embodiment 

or the alternative embodiment is'then added to the original pixel block to result in a pixel 
block corrected for blocking artifacts. This process is repeated for all pixel blocks in the image 
in order to derive a corrected image. 
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It will be apparent that, for various classes of images and particular 
compression methods, the parameters of this method, in particular the block selection 
threshold and the overall weighting factor values, can be optimized to give maximum 
perceptual improvement. 
5 Further, the equations presented above are exemplary of one embodiment of the 

methods described. One of skill in the art will immediately appreciate how their form could be 
improved for computational efficiency in micro-processors and digital signal processors of 
various architectures. For example, the total number of arithmetic operations can be reduced 
by factorizatidn in view of the dimension independence of the 8X8 and 4X4 matrices. Further, 
10 multiplicative operations, including the overall multiplicative weight factor, can be 
implemented by less costly shifts and additions, instead of more costly multiplications. 

Additionally, the particular combination of alternatives chosen from those 
described above can be dictated by a tradeoff of the processing power available in a system 
with the degree of perceptual improvement sought. This invention is adaptable to a range of 
15 degrees of perceptual improvement as increased processing power is available. 

Example 

An example of the functioning of an embodiment of the present invention is 
described herein. Fig. 5 is a 640X480 test image created by expanding each pixel of a 
20 320X240 monochrome source image to four identical neighboring pixels. The source image 
has a luminance varying between 64 and 192 in a total range of 0 to 255 according to the 
following equations. 



luminance = 128 + (64-4R)*cos(4R) 
25 R = (radius in pixel widths/40) 1 ' 5 



Fig. 6 is a DCT transform of this image with all DCT coefficients set to zero 
except for the DC coefficients. It represents a maximally compressed version of Fig. 5 having 
a maximum of blocking artifacts. Each 8X8 pixel block is clearly apparent. 
30 Fig. 7 is a version of Fig. 6 post-processed according to the following early 

embodiment of the present invention. Sub-edge pixel differences A-L were determined as . 
discussed above, resulting in four block-to-block pixel differences for each edge of each 8X8 
pixel block. These coefficients were placed appropriately around the edge of a 4X4 matrix, as 
described for an alternative embodiment of processing operation 7", and were then Hadamard 
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transformed. The overall weighting factor was 0. 125, a very conservative value compared to 
the preferred value of 0375. The upper left hand 3X3 sub-matrix (out of the total 4X4 matrix) 
of Hadamard coefficients, together with the original DC coefficient, were inverse DCT 
. transformed to derive Fig. 7. 
5 Fig. 7 is certainly perceptually much improved, the extreme blocking artifacts 

being much reduced and even eliminated in the central regions of the image. 

The example illustrates the power of the system and method of this invention; to 
achieve substantial reduction in blocking artifacts of image compression algorithms by using 
only simple and computationally inexpensive processing operations. 

10 It should be noted that the above-mentioned embodiments illustrate rather than 

limit the invention, and that those skilled in the art will be able to design many alternative . 
embodiments without departing from the scope of the appended Claims. In the Claims, any 
reference signs placed between parentheses shall not be construed as limiting the Claim. The 
word "comprising" does not exclude the presence of elements or steps other than those listed 

15 in a Claim. The word "a" or "an" preceding an element does not exclude the presences of a 
plurality of such elements. The invention can be implemented by means of hardware 
comprising several distinct elements, and by means of a suitably programmed computer. In the 
device Claim enumerating several means, several of these means can be embodied by one and 
the same item of hardware. 
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1. A method for post-processing a decompressed image, the image having been 

compressed by a process including compression of rectangular blocks of pixels covering the 
original image, said method comprising: 

determining blocking artifact quantities for each pixel block (5) in the 
5 decompressed image that are representative of blocking artifacts, wherein the blocking artifact 
quantities for a pixel block are determined from block-to-block differences between that pixel 
block and pixel blocks adjacent to that pixel block, 

selecting pixel blocks (6) for post-processing according to the block artifact 
quantities for each pixel block, 
10 determining an error correction matrix (7\7 M ) for each selected pixel block from 

the quantities for that selected pixel block, and 

adding the error correction matrices to the selected pixel blocks to derive post- 
processed pixel blocks and the post-processed image. 

15 2. The method of Claim 1 wherein the blocking artifact quantities for a pixel block 

are four quantities determined from the four differences between averages of values of pixels 
along each edge of that pixel block and averages of values of pixels along adjacent edges of 
the adjacent pixel blocks. 

20 3. The method of Claim 1 wherein the blocking artifact quantities for a pixel block 

are determined from averages of differences between the values of pixels of each of two or 
more adjacent pairs of pixels, arid wherein for each pair of pixels one pixel of that pair is at an 
edge of that pixel block and the other pixel of that pair is adjacent at an adjacent edge of the 
adjacent pixel block. 

25 

4. The method of Claim 1 wherein the blocking artifact quantities for a pixel block 

are four quantities determined from the four differences between a zero-frequency (DC) 
transform coefficient of that pixel block and zero-frequency transform coefficients of the four 
orthogonally adjacent pixel blocks. 
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5. The method of Claim 1 wherein a threshold value used in the selecting step is 
of the order of magnitude of errors in pixel values introduced by the combined steps of 
transforming, quantizing, dequantizing, and inverse transforming applied to pixel blocks. 

5 

6. The method of Claim I wherein said selecting selects pixel blocks for which all 
of the four or more quantities are less than a threshold value. 

7. The method of Claim 1 wherein said selecting selects all pixel blocks, and 

10 wherein the method further comprises setting to a threshold value any of the blocking artifact 
quantities for each pixel block which are greater than the threshold value. 

8. The method of Claim 1 wherein each error correction matrix is determined by a 
process comprising linearly interpolating the blocking artifact quantities (7\20) according to 

15 selected spatial configuration weights in order to determine elements of the error correction 
matrices. 

9. The method of Claim 8 wherein said linear interpolation is performed in a 
dimensionally-independent manner according to which a quantity at an edge is interpolated 

20 similarly to all error correction matrix elements that are in a direction perpendicular to that 
edge. 

10. The method of Claim 8 wherein the spatial configuration weights are selected 
such that (i) the largest weight is applied at the edge associated with the quantity to be 

25 interpolated, (ii) the sum of the spatial configuration weights is zero, and (iii) the interpolation 
of equal quantities of opposite sign at two opposite edges result in a linear gradient of error 
correction matrix elements between the two opposite edges. 

11. The method of Claim 10 wherein the pixel blocks have size 8X8 pixels, and 
30 wherein the spatial configuration weights have relative values 44, +2, 0, -1, -2, -2, -1, and 0. 



12. The method of Claim 1 wherein said determining an error correction matrix 

comprises: 
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determining an intermediary error correction matrix (20) having a size smaller 
than the size of the pixel blocks by linearly interpolating the four or more quantities in a 
dimensionally independent manner according to selected spatial configuration weights, 

transforming the intermediary error correction matrix (21 ) to a transform 

5 domain, and 

inverse transforming the transformed intermediary error correction matrix (23) 
to the error correction matrix, wherein for said inverse transformation selected higher order 
transform coefficients are set to zero (22). 

10 13. The method of Claim 1 wherein the error correction matrix is determined so 

that block-to-block pixel differences between two adjacent post-processed pixel blocks are 
smaller than but of the same sign as the block-to-block pixel differences between those two 
adjacent pixel blocks prior to post-processing. 

15 14. A system for post-processing a decompressed image, the image having been 

compressed by a process including compression of rectangular blocks of pixels covering the 
original image, said system comprising: 

means for determining blocking artifact quantities (5) for each pixel block in 
the decompressed image that are representative of blocking artifacts, wherein the blocking 
20 artifact quantities for a pixel block are determined from block-to-block differences between 
that pixel block and pixel blocks adjacent to that pixel block, 

means for selecting pixel blocks for post-processing (6) according to the 
blocking artifact quantities for each pixel block and a threshold value, 

means for determining an error correction matrix (7) for each selected pixel 
25 black from the blocking artifact quantities for that selected pixel block, and 

means for adding (8) the error correction matrices to the selected pixel blocks to 
derive post-processed pixel blocks and the post-processed image. 

15. A system for post-processing a decompressed image/the image having been 

30 compressed by a process including independent compression of non-overlapping rectangular 
blocks of pixels covering the. original image, said system comprising: 

one or more processors ( 1 1 ) for executing program instructions, and 
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one or more memory units (12,13) for storing an image to be processed and 
program instructions, wherein the program instructions cause said one or more processors to 
execute the method of Claim 1. 

5 16. A computer readable media encoded with program instructions for causing one 

or more processors to perform the method of Claim 1. . 

17. A method of reducing blocking artifacts introduced by compression algorithms 

that compress images as blocks of pixels, including the steps of determining block-to-block 
10 differences in edge pixels or in overall intensities between adjacent pixel blocks, selecting 

pixel blocks for post-processing that appear to be in relatively featureless regions of the image, 
interpolating the block-to-block edge differences into an error correction matrix, and then 
subtracting the error correction matrix from the original pixel block. 
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